{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <p style=\"text-align:center\">\n",
    "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
    "        <br>\n",
    "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
    "        |\n",
    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
    "        |\n",
    "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
    "    </p>\n",
    "</center>\n",
    "<h1 align=\"center\">LLM Ops - Tracing, Evaluation, and Analysis</h1>\n",
    "\n",
    "In this tutorial we will learn how to build, observe, evaluate, and analyze a LLM powered application. \n",
    "\n",
    "It has the following sections:\n",
    "\n",
    "1. Understanding LLM-powered applications\n",
    "2. Observing the application using traces\n",
    "3. Evaluating the application using LLM Evals\n",
    "4. Exploring and and troubleshooting the application using UMAP projection and clustering\n",
    "\n",
    "⚠️ This tutorial requires an OpenAI key to run\n",
    "\n",
    "\n",
    "## Understanding LLM powered applications\n",
    "\n",
    "Building software with LLMs, or any machine learning model, is [fundamentally different](https://karpathy.medium.com/software-2-0-a64152b37c35). Rather than compiling source code into binary to run a series of commands, we need to navigate datasets, embeddings, prompts, and parameter weights to generate consistent accurate results. LLM outputs are probabilistic and therefore don't produce the same deterministic outcome every time.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/images/blog/5_steps_of_building_llm_app.png\" width=\"1100\" />\n",
    "\n",
    "There's a lot that can go into building an LLM application, but let's focus on the architecture. Below is a diagram of one possible architecture. In this diagram we see that the application is built around an LLM but that there are many other components that are needed to make the application work.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/images/blog/llm_app_architecture.png\" width=\"1100\" />\n",
    "\n",
    "The complexity that is involved in building an LLM application is why observability is so important. Observability is the ability to understand the internal state of a system by examining its inputs and outputs. Each step of the response generation process needs to be monitored, evaluated and tuned to provide the best possible experience. Not only that, certain tradeoffs might need to be made to optimize for speed, cost, or accuracy. In the context of LLM applications, we need to be able to understand the internal state of the system by examining telemetry data such as LLM Traces.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observing the application using traces\n",
    "\n",
    "LLM Traces and Observability lets us understand the system from the outside, by letting us ask questions about that system without knowing its inner workings. Furthermore, it allows us to easily troubleshoot and handle novel problems (i.e. “unknown unknowns”), and helps us answer the question, “Why is this happening?”\n",
    "\n",
    "LLM Traces are designed to be a category of telemetry data that is used to understand the execution of LLMs and the surrounding application context such as retrieval from vector stores and the usage of external tools such as search engines or APIs. It lets you understand the inner workings of the individual steps your application takes wile also giving you visibility into how your system is running and performing as a whole.\n",
    "\n",
    "Traces are made up of a sequence of `spans`. A span represents a unit of work or operation (think a span of time). It tracks specific operations that a request makes, painting a picture of what happened during the time in which that operation was executed.\n",
    "\n",
    "LLM Tracing supports the following span kinds:\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/images/blog/span_kinds.png\" width=\"1100\"/>\n",
    "\n",
    "\n",
    "By capturing the building blocks of your application while it's running, phoenix can provide a more complete picture of the inner workings of your application. To illustrate this, let's look at an example LLM application and inspect it's traces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Traces and Spans in action\n",
    "\n",
    "Let's build a relatively simple LLM-powered application that will answer questions about Arize AI. This example uses LlamaIndex for RAG and OpenAI as the LLM but you could use any LLM you would like. The details of the application are not important, but the architecture is. Let's get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qq \"arize-phoenix[evals]\" \"llama-index>=0.10.3\" \"openinference-instrumentation-llama-index>=1.0.0\" \"llama-index-callbacks-arize-phoenix>=0.1.2\" \"llama-index-llms-openai\" \"openai>=1\" gcsfs nest_asyncio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
    "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import phoenix as px\n",
    "from llama_index.core import set_global_handler\n",
    "\n",
    "# Setup phoenix tracing\n",
    "px.launch_app()\n",
    "set_global_handler(\"arize_phoenix\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "import phoenix as px\n",
    "from gcsfs import GCSFileSystem\n",
    "from llama_index.core import (\n",
    "    Settings,\n",
    "    StorageContext,\n",
    "    load_index_from_storage,\n",
    ")\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "file_system = GCSFileSystem(project=\"public-assets-275721\")\n",
    "index_path = \"arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/index/\"\n",
    "storage_context = StorageContext.from_defaults(\n",
    "    fs=file_system,\n",
    "    persist_dir=index_path,\n",
    ")\n",
    "\n",
    "Settings.llm = OpenAI(model=\"gpt-4-turbo-preview\")\n",
    "Settings.embed_model = OpenAIEmbedding(model=\"text-embedding-ada-002\")\n",
    "index = load_index_from_storage(\n",
    "    storage_context,\n",
    ")\n",
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an application setup, let's take a look inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "queries = [\n",
    "    \"How can I query for a monitor's status using GraphQL?\",\n",
    "    \"How do I delete a model?\",\n",
    "    \"How much does an enterprise license of Arize cost?\",\n",
    "    \"How do I log a prediction using the python SDK?\",\n",
    "]\n",
    "\n",
    "for query in tqdm(queries):\n",
    "    response = query_engine.query(query)\n",
    "    print(f\"Query: {query}\")\n",
    "    print(f\"Response: {response}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've run the application a few times, let's take a look at the traces in the UI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The Phoenix UI:\", px.active_session().url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The UI will give you an interactive troubleshooting experience. You can sort, filter, and search for traces. You can also view the detail of each trace to better understand what happened during the response generation process.\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/images/trace_details_view.png\" width=\"1100\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to being able to view the traces in the UI, you can also query for traces to use as pandas dataframes in your notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spans_df = px.Client().get_spans_dataframe()\n",
    "spans_df[[\"name\", \"span_kind\", \"attributes.input.value\", \"attributes.retrieval.documents\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With just a few lines of code, we have managed to gain visibility into the inner workings of our application. We can now better understand how things like retrieval, prompts, and parameter weights could be affecting our application. But what can we do with this information? Let's take a look at how to use LLM evals to evaluate our application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (\n",
    "    input(\"The tutorial is about to move on to the evaluation section. Continue [Y/n]?\")\n",
    "    .lower()\n",
    "    .startswith(\"n\")\n",
    "):\n",
    "    assert False, \"notebook stopped\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the application using LLM Evals\n",
    "\n",
    "Evaluation should serve as the primary metric for assessing your application. It determines whether the app will produce accurate responses based on the data sources and range of queries.\n",
    "\n",
    "While it's beneficial to examine individual queries and responses, this approach is impractical as the volume of edge-cases and failures increases. Instead, it's more effective to establish a suite of metrics and automated evaluations. These tools can provide insights into overall system performance and can identify specific areas that may require scrutiny.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first convert our traces into a workable datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents\n",
    "\n",
    "retrieved_documents_df = get_retrieved_documents(px.active_session())\n",
    "queries_df = get_qa_with_reference(px.active_session())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use Phoenix's LLM Evals to evaluate these queries. LLM Evals uses an LLM to grade your application based on different criteria. For this example we will use the evals library to see if any `hallucinations` are present and if the `Q&A Correctness` is good (whether or not the application answers the question correctly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "from phoenix.evals import (\n",
    "    HALLUCINATION_PROMPT_RAILS_MAP,\n",
    "    HALLUCINATION_PROMPT_TEMPLATE,\n",
    "    QA_PROMPT_RAILS_MAP,\n",
    "    QA_PROMPT_TEMPLATE,\n",
    "    OpenAIModel,\n",
    "    llm_classify,\n",
    ")\n",
    "\n",
    "nest_asyncio.apply()  # Speeds up OpenAI API calls\n",
    "\n",
    "# Check if the application has any indications of hallucinations\n",
    "hallucination_eval = llm_classify(\n",
    "    dataframe=queries_df,\n",
    "    model=OpenAIModel(model=\"gpt-4-turbo-preview\", temperature=0.0),\n",
    "    template=HALLUCINATION_PROMPT_TEMPLATE,\n",
    "    rails=list(HALLUCINATION_PROMPT_RAILS_MAP.values()),\n",
    "    provide_explanation=True,  # Makes the LLM explain its reasoning\n",
    ")\n",
    "hallucination_eval[\"score\"] = (\n",
    "    hallucination_eval.label[~hallucination_eval.label.isna()] == \"factual\"\n",
    ").astype(int)\n",
    "\n",
    "# Check if the application is answering questions correctly\n",
    "qa_correctness_eval = llm_classify(\n",
    "    dataframe=queries_df,\n",
    "    model=OpenAIModel(model_name=\"gpt-4-turbo-preview\", temperature=0.0),\n",
    "    template=QA_PROMPT_TEMPLATE,\n",
    "    rails=list(QA_PROMPT_RAILS_MAP.values()),\n",
    "    provide_explanation=True,  # Makes the LLM explain its reasoning\n",
    "    concurrency=4,\n",
    ")\n",
    "\n",
    "qa_correctness_eval[\"score\"] = (\n",
    "    hallucination_eval.label[~qa_correctness_eval.label.isna()] == \"correct\"\n",
    ").astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hallucination_eval.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_correctness_eval.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from the results, one of the queries was flagged as a hallucination. Let's log these results to the phoenix server to view the hallucinations in the UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.trace import SpanEvaluations\n",
    "\n",
    "px.Client().log_evaluations(\n",
    "    SpanEvaluations(eval_name=\"Hallucination\", dataframe=hallucination_eval),\n",
    "    SpanEvaluations(eval_name=\"QA Correctness\", dataframe=qa_correctness_eval),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the hallucinations logged, let's take a look at them in the UI. You will notice that the traces that correspond to hallucinations are clearly marked in the UI and you can now query for them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The Phoenix UI:\", px.active_session().url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now identified that there are certain queries that are resulting in hallucinations or incorrect answers. Let's see if we can use LLM Evals to identify if these issues are caused by the retrieval process for RAG. We are going to use an LLM to grade whether or not the chunks retrieved are relevant to the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.experimental.evals import (\n",
    "    RAG_RELEVANCY_PROMPT_RAILS_MAP,\n",
    "    RAG_RELEVANCY_PROMPT_TEMPLATE,\n",
    "    OpenAIModel,\n",
    "    llm_classify,\n",
    ")\n",
    "\n",
    "retrieved_documents_eval = llm_classify(\n",
    "    dataframe=retrieved_documents_df,\n",
    "    model=OpenAIModel(model=\"gpt-4-turbo-preview\", temperature=0.0),\n",
    "    template=RAG_RELEVANCY_PROMPT_TEMPLATE,\n",
    "    rails=list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values()),\n",
    "    provide_explanation=True,\n",
    ")\n",
    "\n",
    "retrieved_documents_eval[\"score\"] = (\n",
    "    retrieved_documents_eval.label[~retrieved_documents_eval.label.isna()] == \"relevant\"\n",
    ").astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_documents_eval.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like we are getting a lot of irrelevant chunks of text that might be polluting the prompt sent to the LLM. Let's log these evals to phoenix, at which point phoenix will automatically calculate retrieval metrics for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.trace import DocumentEvaluations\n",
    "\n",
    "px.Client().log_evaluations(\n",
    "    DocumentEvaluations(eval_name=\"Relevance\", dataframe=retrieved_documents_eval)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we once again visit the UI, we will now see that Phoenix has aggregated up retrieval metrics (`precision`, `ndcg`, and `hit`). We see that our hallucinations and incorrect answers directly correlate to bad retrieval!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The Phoenix UI:\", px.active_session().url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/images/screenshots/document_evals_on_traces.png\" />\n",
    "\n",
    "There are many more evaluations metrics that can be used to make determinations about the quality of your application. Phoenix supports evaluating not only traces but individual spans (such as retrievers) as well as performing retrieval analysis for your document chunks. Evaluation metrics can also be customized for your specific use-case. For more details, consult the phoenix docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (\n",
    "    input(\"The tutorial is about to move on to the analysis section. Continue [Y/n]?\")\n",
    "    .lower()\n",
    "    .startswith(\"n\")\n",
    "):\n",
    "    assert False, \"notebook stopped\"\n",
    "\n",
    "px.close_app()  # Close the Phoenix UI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring and and troubleshooting the application using UMAP projection and clustering\n",
    "\n",
    "We have so far figured out how to trace our application and how to evaluate it. But what if we need to understand the application at a higher level? What if we need to understand if the application is performing badly for a certain topic or if there are any patterns in the data that we can use to improve the application? This is where UMAP projection and clustering comes in. UMAP and clustering let's you view the query embeddings of your application in a 3D space. This allows you to see patterns in the data and understand how your application is performing in various clusters (semantic groups).\n",
    "\n",
    "Since we only ran the application for 4 queries, we won't be able to see any patterns in the data. Let's pull in a larger dataset of queries and run the application again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Pull in queries from the LLM\n",
    "query_df = pd.read_parquet(\n",
    "    \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet\",\n",
    ")\n",
    "\n",
    "query_ds = px.Dataset.from_open_inference(query_df)\n",
    "\n",
    "query_ds.dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also pull in the embeddings for our knowledge base. This will allow us to see how how document chunks are being retrieved by the application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "def storage_context_to_dataframe(storage_context: StorageContext) -> pd.DataFrame:\n",
    "    \"\"\"Converts the storage context to a pandas dataframe.\n",
    "\n",
    "    Args:\n",
    "        storage_context (StorageContext): Storage context containing the index\n",
    "        data.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: The dataframe containing the index data.\n",
    "    \"\"\"\n",
    "    document_ids = []\n",
    "    document_texts = []\n",
    "    document_embeddings = []\n",
    "    docstore = storage_context.docstore\n",
    "    vector_store = storage_context.vector_store\n",
    "    for node_id, node in docstore.docs.items():\n",
    "        document_ids.append(node.hash)  # use node hash as the document ID\n",
    "        document_texts.append(node.text)\n",
    "        document_embeddings.append(np.array(vector_store.get(node_id)))\n",
    "    return pd.DataFrame(\n",
    "        {\n",
    "            \"document_id\": document_ids,\n",
    "            \"text\": document_texts,\n",
    "            \"text_vector\": document_embeddings,\n",
    "        }\n",
    "    )\n",
    "\n",
    "\n",
    "database_df = storage_context_to_dataframe(storage_context)\n",
    "database_df = database_df.drop_duplicates(subset=[\"text\"])\n",
    "database_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now launch Phoenix with these two datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a random sample of 500 documents (including retrieved documents)\n",
    "# this will be handled by by the application in a coming release\n",
    "num_sampled_point = 500\n",
    "retrieved_document_ids = set(\n",
    "    [\n",
    "        doc_id\n",
    "        for doc_ids in query_df[\":feature.[str].retrieved_document_ids:prompt\"].to_list()\n",
    "        for doc_id in doc_ids\n",
    "    ]\n",
    ")\n",
    "retrieved_document_mask = database_df[\"document_id\"].isin(retrieved_document_ids)\n",
    "num_retrieved_documents = len(retrieved_document_ids)\n",
    "num_additional_samples = num_sampled_point - num_retrieved_documents\n",
    "unretrieved_document_mask = ~retrieved_document_mask\n",
    "sampled_unretrieved_document_ids = set(\n",
    "    database_df[unretrieved_document_mask][\"document_id\"]\n",
    "    .sample(n=num_additional_samples, random_state=0)\n",
    "    .to_list()\n",
    ")\n",
    "sampled_unretrieved_document_mask = database_df[\"document_id\"].isin(\n",
    "    sampled_unretrieved_document_ids\n",
    ")\n",
    "sampled_document_mask = retrieved_document_mask | sampled_unretrieved_document_mask\n",
    "sampled_database_df = database_df[sampled_document_mask]\n",
    "\n",
    "database_schema = px.Schema(\n",
    "    prediction_id_column_name=\"document_id\",\n",
    "    prompt_column_names=px.EmbeddingColumnNames(\n",
    "        vector_column_name=\"text_vector\",\n",
    "        raw_data_column_name=\"text\",\n",
    "    ),\n",
    ")\n",
    "database_ds = px.Dataset(\n",
    "    dataframe=sampled_database_df,\n",
    "    schema=database_schema,\n",
    "    name=\"database\",\n",
    ")\n",
    "\n",
    "session = px.launch_app(primary=query_ds, corpus=database_ds, run_in_thread=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Phoenix's embedding projection feature, we can now view the query embeddings in a 3D space. What you are looking at is a point-cloud where each point represents a query. When embeddings are projected into a lower dimensional space, UMAP preserves the semantic distance that the embeddings encode. This means that the points that are closer together are more similar. This allows us to see patterns in the data and understand how our application is performing in various clusters. If you click through the clusters, you will see clusters of queries that are significantly farther away from the knowledge base. This means that the knowledge base does not contain enough information to answer the user's question (In this case, the document chunks don't contain any information about pricing. Can you find this cluster?). \n",
    "\n",
    "Phoenix's embedding view also supports coloring and cluster metrics by evaluations! This means you can use it to find pockets of poor user feedback or hallucinations. Phoenix's embedding view supports embeddings for text, images, and video so a most forms of unstructured data can be analyzed.\n",
    "\n",
    "Last but not least, Phoenix can draw the connections between queries and the document chunks. This allows you to see which document chunks are being retrieved for each query and to visualize the semantic distance between them. \n",
    "\n",
    "<img src=\"https://github.com/Arize-ai/phoenix-assets/raw/main/gifs/corpus_search_and_retrieval.gif?raw=true\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Phoenix is a powerful companion for building LLM-powered applications. It allows you to observe, evaluate, and explore your application at every step of you development process. For more details, consult the [phoenix docs](https://docs.arize.com/phoenix)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
